A method for color naming and description of color composition in images

Color is one of the main visual cues and has been frequently used in image processing, analysis and retrieval. The extraction of highlevel color descriptors is an increasingly important problem, as these descriptions often provide link to image content. When combined with image segmentation color naming can be used to select objects by color, describe the appearance of the image and even generate semantic annotations. For example, regions labeled as light blue and strong green may represent sky and grass, vivid colors are typically found in man-made objects, and modifiers such as brownish, grayish and dark convey the impression of the atmosphere in the scene. This paper presents a computational model for color categorization, naming and extraction of color composition. In this work we start from the National Bureau of Standards’ recommendation for color names [4], and through subjective experiments develop our color vocabulary and syntax. Next, to attach the color name to an arbitrary input color, we design a perceptually based color naming metric. Finally, we extend the method and develop a scheme for extracting the color composition of a complex image. The algorithm follows the relevant neurophysiological findings and studies on human color categorization. In testing the method the known color regions in different color spaces were identified accurately, the color names assigned to randomly selected colors agreed with human judgments, and the color composition extracted from natural images was consistent with human observations. 1. COLOR PERCEPTION AND CATEGORIZATION Color is a perceptual phenomenon related in a complex way to the spectral characteristics of electro-magnetic radiation that strikes the retina [1]. According to the theories postulated to explain human perception, color vision is initiated in retina where the three types of cones receive the light stimulus. The cone responses are then coded into one achromatic and two antagonistic chromatic signals. These signals are interpreted in the cortex, in the context of other visual information received at the same time and the previously accumulated visual experience. Once the intrinsic character of colored surface has been represented internally, one may think that the color processing is complete. However, humans go beyond the purely perceptual experience to categorize things and attach linguistic labels to them and color is no exception. A breakthrough in the current understanding of color categorization came from a study conducted by Berlin and Kay [2] who discovered remarkable regularities in the shape of the basic color vocabulary across different languages. Berlin and Kay introduced a concept of basic color terms and identified 11 basic terms in English (black, white, red, green, yellow, blue, brown, pink, orange, purple and gray). They also established the definitions of the focal colors as the centers of color categories and graded/fuzzy membership. Many later studies have proven this hypothesis, indicating that prototypical colors play a crucial role in internal representation of color categories, and the membership in categories is judged relative to the prototype. Unfortunately, the mechanism of color naming is still not completely understood: The existing theories of color naming have important drawbacks and are not implemented as full-fledged computational models [3]. Although color spaces allow us to specify or describe colors in unambiguous manner, in everyday life we mainly identify colors by their names. Hence, there were several attempts towards designing a vocabulary, syntax and standard method for choosing color names. Following the recommendation of the Inter-Society Council the National Bureau of Standards developed the ISCC-NBS lexicon of color names for 267 regions in color space [4]. This lexicon employs English terms to describe colors along the dimensions of hue (28 names constructed from a basic set red, orange, yellow, green, blue, violet, purple, pink, brown, olive, black, white and gray), lightness (very dark, dark, medium, light and very light), saturation (grayish, moderate, strong and vivid), and lightness/saturation (brilliant, pale and deep). One problem with this model is the lack of systematic syntax, which was addressed during the design of a new Color-Naming System (CNS) [5]. Both methods are based on the Munsell system [1], and thus provide explanation on how to locate each name within the Munsell color space. However, a notable disadvantage of the Munsell system for color-based processing is the lack of the exact transform from other color spaces. Furthermore, it is not obvious how to use the ISCCNBS lexicon or CNS syntax to automatically attach names to sample colors, describe the color regions in a scene, and ultimately, communicate the color composition of an image. The only computational model that provides the solution to these problems is proposed in [3]. This work starts from the Berlin and Kay’s color naming data and applies the normal distribution as a category model. However, the method is constrained to the fundamental level of color naming, as it was fitted to the basic color names, and does not account for commonly used saturation or luminance modifiers. Since it depends on the intricate fitting procedure, there is no straightforward extension of the model to include these attributes. The goal of our work is to develop a perceptually based color naming method that allows for higher level of color communication. The method should satisfy the following properties. Color naming operation should be performed in a perceptually controlled way so that the names attached to different colors reflect perceived color differences among them. Segmenting the color space into the color categories should produce smooth regions. The method should account for the basic color terms and use systematic syntax to combine them. It should respect the graded nature of category membership, the universality of color foci, and produce results in agreement with human judgments. The first step in our work (Section 2) involves a development of balanced and wellrepresented set of color prototypes, i.e. vocabulary, and the corresponding syntax. In the next step we design a color naming metric, which, for an arbitrary input color, determines the category membership (Section 3). Finally, we extend this approach to name color regions and develop descriptors of color composition in complex scenes (Section 4). 2. THE COLOR NAMING VOCABULARY AND SYNTAX As a starting point for our vocabulary we adopted the ISCCNBS lexicon, since it provides a model developed using controlled perceptual experiments and includes the basic color terms. Each category is represented with its centroid color, thus preserving the notion of color foci. Yet, due to the strict naming convention the dictionary includes several names that are not well understood by the general public (blackish red for example) and lacks a systematic syntax. As the centroid colors span the color space in uniform fashion and allow grading between the categories, we decided to use these points as the prototypes in our color naming algorithm, yet we had to devise our own name structure that follows few simple systematic rules. To determine a reliable color vocabulary, we have performed a set of subjective experiments aimed at testing the agreement between the names from the ISCC-NBS lexicon and human judgments, adjusting the lexicon for the use in automatic color naming applications, and, most importantly, gain better understanding of human color categorization and naming. We conducted four experiments: color listing experiment aimed at testing eleven basic color categories established in Berlin and Kay study [2], color composition experiment aimed at determining color vocabulary used in describing complex scenes, and two color naming experiments aimed at understanding human behavior in color naming and adjusting the differences between the human judgments and the semantics of the ISCC-NBS vocabulary. Ten subjects participated in the experiments. All subjects had normal color vision and normal or corrected-to-normal vision. Color listing experiment In addition to the 11 basic color terms in English, some studies indicated few marginal cases such as beige/tan or olive and violet. To test the relevance of these terms we asked each of our subjects to write on a sheet of paper names of at least twelve “most important” colors. Color composition experiment Here the subjects were presented with 40 photographic images in a sequence and asked to name all the colors in the image. The images were selected to include different color compositions, spatial frequencies and arrangements among the colors, and provided broad content. Each image was displayed on the computer monitor, against light gray background. The order of presentation was randomly generated for each subject. Color naming experiments In these experiments, the subjects were presented with the 267 centroid colors and asked to name each one of them. The color patches were displayed on the computer monitor, in a room that received daylight illumination. The monitor was calibrated so that there was no difference between the colors on the monitor and corresponding chips form the Munsell Book of Colors [1]. In the first experiment, 64×64 pixel patches were arranged into 9×6 matrix and displayed against the light gray background. The names were assigned by typing into a text box below each patch. After naming all the colors, the display was updated with the new set of patches, until all 267 centroid colors have been named. In the second color naming experiments, only one 200×200 pixels color patch was displayed on the screen. In all experiments subjects were advised to use common color names and common modifiers for brightness or saturation, to avoid names derived from objects/materials, and derive new names by combining the basic hues with the modifier –ish (greenish blue). Here is the brief summary of the most important findings. As expected, in the Color listing experiment 11 basic colors were found on the list of every subject. Nine subjects included beige, four included violet, and two included cyan and magenta. Modifiers for hue, saturation and luminance were not used. None of the subjects listed more than 14 color names. Surprisingly, the subjects maintained the same limited vocabulary when describing images in the Color composition experiment, and added only beige to the basic colors. In attempt to distinguish between the different types of the same hue, the subjects often used modifiers for saturation and luminance. The modifiers for hue were not frequently used. Although most of the images had rich color histograms, the subjects were not able to perceive more than ten colors at once. Dark colors, which typically exist in natural images due to shadows and edges, or light colors due to highlights and specularities, were never included in the description, and were named only when referring to well-defined objects/regions. The subjects showed the highest level of precision in the Color naming experiments. Most of them (9/10) frequently used modifiers for hue, saturation or brightness. Seven subjects used olive, although they had not used this term in the previous experiments. On the other hand, although it had been listed in the Color listing experiment, violet was seldom used and was most of the time described as bluish purple. Modifiers brilliant and deep, as in the ISCC-NBS vocabulary, were not used they were replaced with the descriptors strong light and strong dark. There was a very good degree of concordance between the subjects; In the first Color naming experiment, 223 samples were assigned the same hue by all subjects (with variations in the use of modifiers), 15 were assigned into one of two related hue categories (such as greenish blue and blue), 19 were assigned into one of three related hue categories. The remaining 10 color samples were not reliably assigned into any category. Out of 223 hues that were assigned into the same category by all subjects, 195 were the same as in the ISCC-NBS vocabulary, 22 were assigned to a related hue, and 8 were assigned entirely different color name. Similar results were obtained in the second Color naming experiment. The major difference between the subjective results and the color names from the ISCC-NBS vocabulary involved the use of the saturation modifiers colors appeared less saturated to our subjects and they generally applied higher thresholds when attaching modifiers like vivid or grayish. There was also a very good agreement between the two experiments 79 samples were assigned the same color name in both experiments, 121 were assigned the same hue, 42 were assigned one of two related hues, 20 were assigned one of three related hues, and 5 samples were assigned into non-related categories. The difference between the two experiments was in the use of modifiers, since the same color was often perceived lighter and more chromatic when displayed in the smaller window. For the final vocabulary we have selected the names from the first Color naming experiment, as they were generated in interactions with multiple colors and seem to provide a better representative of the real-world applications. Based on our findings we have devised the final vocabulary and generalized it in the following syntax (note that : denotes “is defined as”, | denotes meta-or, and [] is the optional occurrence of the enclosed construct): color name : achromatic name | chromatic name achromatic name : lightness gray | black | white chromatic name : lightness saturation hue | saturation lightness hue lightness : blackish | very dark | dark | medium | light | very light | whitish saturation : grayish | moderate | medium | strong | vivid hue : generic hue | -ish form generic hue generic hue : red | orange | brown | yellow | green | blue | purple | pink | beige | olive ish form : reddish | brownish | yellowish | greenish | bluish | purplish |